unicode_data refactors
#147622
Open
unicode_data refactors
#147622
+6,191
−1,584
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
If you want to modify |
rustbot
added
S-waiting-on-review
Kmeakin
force-pushed
the
km/unicode-data/refactors
branch
from
2c5244e to
90adbe2
Compare
bjorn3
reviewed
Oct 13, 2025
Kmeakin
force-pushed
the
km/unicode-data/refactors
branch
from
90adbe2 to
1a646cf
Compare
This comment has been minimized.
This comment has been minimized.
joboet
reviewed
Oct 14, 2025
| } | ||
|
|
||
| fn rustfmt(path: &str) { | ||
| std::process::Command::new("rustfmt").arg(path).status().expect("rustfmt failed"); |
There was a problem hiding this comment.
Is rustfmt really always in PATH when this command is run? Otherwise, I think it'd be easier to slap a big #[rustfmt::skip] on the mod unicode_data.
There was a problem hiding this comment.
The intention is to keep the generated unicode_data.rs readable without having to carefully construct well-formatted code in the metaprogram. For that, we need to rust rustfmt
Kmeakin
force-pushed
the
km/unicode-data/refactors
branch
3 times, most recently
from
1b56c98 to
dc0dcf5
Compare
rustbot
added
the
T-bootstrap
|
This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed. Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers. |
This comment has been minimized.
This comment has been minimized.
Instead of `include_str!()`ing `range_search.rs`, just make it a normal module under `core::unicode`. This means the same source code doesn't have to be checked in twice, and it plays nicer with IDEs. Also rename it to `rt` since it includes functions for searching the bitsets and case conversion tables as well as the range represesentation.
Remove `#[rustfmt::skip]` from all the generated modules in `unicode_data.rs`. This means we won't have to worry so much about getting indetation and formatting right when generating code. Exempted for now some tables which would be too big when formatted by `rustfmt`.
This check was made redundant (it will always be true) when we removed all ASCII characters from the tables (rust-lang@a8c6694).
To make the final output code easier to see: * Get rid of the unnecessary line-noise of `.unwrap()`ing calls to `write!()` by moving the `.unwrap()` into a macro. * Join consecutive `write!()` calls using a single multiline format string. * Replace `.push()` and `.push_str(format!())` with `write!()`. * If after doing all of the above, there is only a single `write!()` call in the function, just construct the string directly with `format!()`.
Kmeakin
force-pushed
the
km/unicode-data/refactors
branch
from
dc0dcf5 to
d2c9773
Compare
This comment has been minimized.
This comment has been minimized.
Instead of generating a standalone executable to test `unicode_data`, generate normal tests in `coretests`. This ensures tests are always generated, and will be run as part of the normal testsuite. Also change the generated tests to loop over lookup tables, rather than generating a separate `assert_eq!()` statement for every codepoint. The old approach produced a massive (20,000 lines plus) file which took minutes to compile!
Kmeakin
force-pushed
the
km/unicode-data/refactors
branch
from
d2c9773 to
41d988f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Minor refactors to
unicode_datathat occured to me while trying to reduce the size of the tables. Splitting into a separate PR. NFC